Companies and individuals seeking to reduce the expense and overhead associated with maintaining their own computing resources have turned instead to purchasing remote computing services, such as remote program execution over multiple virtual machine instances and remote data storage, offered by computing resource service providers to customers. Such remote computing services are often configurable and scalable to meet various computing needs of the customers of the computing resource service provider.
In a large distributed computing system (e.g., multiple distributed data centers) of a computing resource service provider, various customers, users, services, and resources of the computing resource service provider are in frequent communication with each other. Balancing customer access to the computing resources and keeping the system highly available becomes more challenging as the complexity and usage of the system increases. In such distributed computing systems, it can be difficult to isolate and troubleshoot issues within the system, and even when issues are identified, reliably distributing the resources of the system may be challenging. Furthermore, these distributed computing systems may be leveraged to execute large-scale applications or jobs. Along with executing large scale applications or jobs, the computing resources in these environments may be replaced, new hardware may be added, the configuration of the computing resources may be modified, or individual computing resources may fail and need to be replaced. These changes to the distributed computing systems may pose a variety of problems to computing resource service providers.